Video acquisition with processing based on ancillary data

ABSTRACT

Systems and techniques for processing sequences of video images involve receiving, on a computer, data corresponding to a sequence of video images detected by an image sensor. The received data is processed using a graphics processor to adjust one or more visual characteristics of the video images corresponding to the received data. The received data can include video data defining pixel values and ancillary data relating to settings on the image sensor. The video data can be processed in accordance with ancillary data to adjust the visual characteristics, which can include filtering the images, blending images, and/or other processing operations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. §120 of U.S. patentapplication No. 12/772,826, entitled “Video Acquisition with IntegratedGPU Processing,” filed May 3, 2010, which is a continuation of U.S.patent application No. 11/239,034, entitled “Video Acquisition withIntegrated GPU Processing,” filed Sep. 29, 2005, which is now U.S. Pat.No. 7,711,200, both of which are incorporated herein by reference intheir entirety.

TECHNICAL FIELD

This description relates to processing video images, and moreparticularly to acquisition of video data with integrated processingusing a graphics processing unit.

BACKGROUND

Image sensors can be used to detect video sequences that can be used byvarious client processes in a laptop or desktop computer environment.For example, detected video sequences can be used for webcamapplications, videophone operations, or video editing. Conventionalcameras for use with computers are external peripherals, althoughbuilt-in cameras can also be used. Such cameras can use charge-coupleddevice (CCD) sensors or complementary metal-oxide semiconductor (CMOS)digital image sensors to detect images. CCD sensors are commonly used indigital still and video cameras and provide a relatively high qualityimage. CMOS sensors generally require less power and are less expensivebut provide a lower quality image than CCD sensors, especially at lowlight levels.

SUMMARY

Techniques and systems can be implemented to improve the quality ofimages detected by image sensors to improve the quality of the detectedimages. In some implementations, the described techniques and systemscan be used to enable CMOS sensors to be used in place of relativelyexpensive CCD sensors, such as to achieve comparable quality from a CMOSsensor as from a CCD sensor. In particular, processing can be performedusing software, hardware, or a combination of the two to filter outnoise, which is increasingly present as light levels decrease; increasedynamic range; improve the overall image quality and/or color fidelity;and/or perform other image processing. Such processing can be offloaded,at least in part, from a computer's central processing unit (CPU) to agraphics processing unit (GPU) to provide a more efficient use ofresources and to support adequate frame rates and image resolutions.

In one general aspect, data corresponding to a sequence of video imagesdetected by an image sensor is received. The received data is processedusing a graphics processor to adjust one or more visual characteristicsof one or more of the video images corresponding to the received data.

Implementations can include one or more of the following features. Thesequence of video images includes image data received at about thirtyframes per second. The received data includes pixel values for each ofmultiple pixels in each video image. The graphics processor is aprocessor having a vector-based instruction set and that automaticallyperforms multiple iterations of one or more instructions on multipledata items. The processed data is forwarded to a central processing unitof the computer for use by a client process running on the centralprocessing unit. The central processing unit performs one of compressionor encoding of the processed data. The one or more visualcharacteristics include noise reduction, color correction, scaling,sharpness, and/or color calibration.

A first pixel shading program is used to calculate filtered motion datafor a frame, and a second pixel shading program is used to calculatetemporal-filtered pixel data for the frame. Two buffers associated withthe graphics processor are provided for use in storing temporal-filteredpixel data for successive frames, and two buffers associated with thegraphics processor are provided for use in storing filtered motion datafor the successive frames. Each buffer stores input data or output datafor one of the successive frames and stores the other of input data oroutput data for another of the successive frames. Temporal filteringperformed by the second pixel shading program is based at least in parton the filtered motion data and a luminance value for areas of theframe.

Processing the received data involves performing gamma correction on oneor more of the video images based on the luminance of an area of one ormore of the video images. Alternatively or in addition, performing gammacorrection involves applying a single-channel luminance modification torelatively dark areas of the one or more video images and applying amulti-channel luminance modification to other areas of the one or morevideo images. As another alternative, performing gamma correction caninvolve determining a single-channel gamma correction and amulti-channel gamma correction and interpolating the single-channelgamma correction and the multi-channel gamma correction using a linearor non-linear interpolation function. The linear or non-linearinterpolation function can include an offset to increase a contributionof the multi-channel gamma correction.

Processing the received data can involve detecting motion between aparticular frame and a preceding frame in the sequence of video imagesand filtering the particular frame using the preceding frame based atleast in part on detected motion. Filtering the particular frame canalso be based at least in part on an estimated noise level for theparticular frame and/or on a luminance value of pixels in the particularframe. Processing the received data can include filtering motion data,such as by assigning a motion value for each pixel according to amaximum motion of the pixel and at least one adjacent pixel.

Processing the received data can include alternately generating motiondata for a frame using motion data for a preceding frame stored in afirst texture buffer and storing the motion data for the frame in asecond texture buffer. Motion data for a subsequent frame can begenerated using the motion data stored in the second texture buffer, andthe motion data for the subsequent frame can be stored in the firsttexture buffer. Processing the received data can further includealternately filtering pixel data for a frame using filtered pixel datafor a preceding frame stored in a third texture buffer and filteringpixel data for a subsequent frame using the filtered pixel data storedin the fourth texture buffer. The filtered pixel data for the frame canbe stored in a fourth texture buffer, and the filtered pixel data forthe subsequent frame can be stored in the third texture buffer. Themotion data can include a value indicating an estimated degree of motionbetween frames and a value indicating a filtering strength. Processingthe received data can include filtering the particular frame using thepreceding frame based at least in part on an estimated noise level forthe particular frame. The received data is processed using bothsingle-channel gamma correction and multi-channel gamma correction. Apolynomial function is used to interpolate the single-channel gammacorrection and the multi-channel gamma correction according to aluminance value.

In another general aspect, an image processing system includes aprocessor and a graphics processor. The processor is operable to receivedata defining to a sequence of video images from an image sensor. Thegraphics processor is operable to process the received data to adjustone or more visual characteristics of one or more of the video imagesdefined by the received data.

Implementations can include one or more of the following features. Theprocessor is further operable to preprocess the received data beforesending the received data to the graphics processor. The processorreceives processed data from the graphics processor for use by a clientapplication. The graphics processor includes at least a first texturebuffer and a second texture buffer and is operable to alternatelyprocess received data for a frame using processed data from a precedingframe stored in the first texture buffer and process received data for aframe using processed data from a preceding frame stored in the secondtexture buffer. Processing the received data using processed data storedin the first texture buffer generates new processed data that is storedin the second texture buffer, and processing the received data usingprocessed data stored in the second texture buffer generates newprocessed data that is stored in the first texture buffer. The graphicsprocessor processes the received data based on a detected amount ofmotion between successive frames and an estimated noise level for eachparticular frame.

In another general aspect, video data detected by an image sensor anddefining a sequence of frames is received. Ancillary data indicating oneor more characteristics ancillary to the video data is also received.The video data is processed based, at least in part, on the ancillarydata to produce converted frames.

Implementations can include one or more of the following features. Theancillary data includes a temperature in a vicinity of the image sensor,and processing the video data comprises adjusting one or more parametersor algorithm based on the temperature. Alternatively or in addition, theancillary data includes settings data for the image sensor. The settingsdata indicates one or more adjustable setting of the image sensor usedfor sensing at least a portion of the video data. The video data isreceived in packets that include video data and associated settingsdata. Multiple packets are used to transfer each frame of video data.The packets are buffered until at least one complete frame is received.The settings data is provided in response to a query from a videodigitizer.

Processing the video data involves processing video image frames toperform temporal filtering. Processing frames to reduce noise involvesincreasing a contribution from at least one preceding frame. A strengthof filtering is determined for each frame based on the settings data.Processing the video data involves determining when to change modes forperforming dynamic range expansion. The settings data one or more gainlevels, a current luminance, an average luminance, and/or one or moremodes.

In another general aspect, data defining a sequence of frames detectedby an image sensor is received. The received data includes video datadefining color values and luminance values for each frame and ancillarydata defining one or more conditions relating to the detected frames. Adegree of motion between a frame of the sequence of frames and one ormore preceding frames is determined for one or more areas of the frame.Video data for the frame is filtered by adjusting the video data usingvideo data from one or more preceding frames based, at least in part, onthe degree of motion, the ancillary data, and the luminance values.

Implementations can include one or more of the following features. Thevideo data defines color values and luminance values for each ofmultiple pixels, and a degree of motion is determined using a maximumdetected motion among each pixel and at least four adjacent pixels. Thevideo data from one or more preceding frames includes filtered videodata for the preceding frame. The video data includes motion data.Alternatively or in addition, the video data defines color values andluminance values for each of multiple pixels and the filtered motiondata for the frame comprises a detected motion for each pixel adjustedaccording to a degree of motion for each pixel of the preceding frameand a filtering parameter for each pixel of the preceding frame. Thefiltering parameter is based, at least in part, on an estimated level ofnoise for each pixel of the preceding frame. The estimated level ofnoise is determined according to the ancillary data and the luminancevalues, and the ancillary data includes settings data received from theimage sensor. Each frame includes multiple pixels and the video dataincludes pixel values. The filtered video data for the frame iscalculated using a pixel value for each pixel, a filtered pixel valuefor each pixel of the preceding frame, and a filtering parameter foreach pixel of the frame. The filtering parameter is based, at least inpart, on an estimated level of noise for each pixel of the precedingframe, and the estimated level of noise is determined according to theancillary data and the luminance values. The filtering parameter foreach pixel is used to adjust each of the pixel value and the filteredpixel value of the preceding frame in calculating the filtered videodata.

In another general aspect, a sequence of frames is detected by an imagesensor. The frames are detected using two or more different gain levels.A frame is blended with one or more other frames detected using adifferent gain level to produce a blended frame for replacing the framein the sequence of frames.

Implementations can include one or more of the following features. Thesequence of frames is a portion of a video sequence. The frame includesmultiple pixel values and blending the frame with one or more otherframes involves normalizing each pixel value for the frame and the oneor more other frames, adjusting the normalized pixel values usingblending factors, and combining the adjusted, normalized pixel values togenerate a blended pixel value. Normalizing each pixel value involvescompensating for the gain level used to detect the pixel value. A degreeof motion between frames in the sequence of frames is estimated for eachof multiple pixels in a particular frame. It is determined whether thedegree of motion for each pixel exceeds a threshold, and, if so,compensation for the gain level used to detect the pixel is performed toproduce a replacement pixel for replacing the pixel in the particularframe. Detecting motion between frames involves comparing correspondingpixels in successive frames detected using a particular gain level.Compensation for the gain level of successive frames in the sequence offrames can also be performed, and detecting motion between frames caninvolve comparing corresponding pixels in gain-compensated, successiveframes. Temporal filtering of the blended frames is performed.

Pixel data for a subsequent frame in the video sequence is received, andthe pixel data for the subsequent frame is blended with the pixel datafrom a subset of the preceding frames to generate a subsequent blendedframe for the video sequence. Motion between frames is detected based onthe pixel data. Blending of pixel data is performed only for pixelshaving motion below a selected threshold. Compensation for gain levelsassociated with pixels having motion above the selected threshold isperformed. Motion is detected between one of the frames and a precedingframe with a same gain level or between one of the frames and apreceding frame with a different gain level. The pixel data is blendedusing a weighted combination of the pixel data. Each pixel of theblended frame includes a combination of weighted pixel data terms, andeach pixel data term is weighted using a blending factor correspondingto a luminance value of the pixel. Multiple temporally spaced blendedframes are filtered to reduce noise. Motion is detected for use indetermining whether to blend frames and for use in determining astrength of temporal filtering.

In yet another general aspect, a system for expanding the dynamic rangeof a video sequence includes a memory for storing a sequence of framesdetected by an image sensor. The frames are detected using two or moredifferent gain levels. A first module is operable to detect motionbetween frames in the sequence, and a second module is operable to blendframes in the sequence to produce processed frames with an expandeddynamic range. A degree of blending is based on whether motion isdetected.

Implementations can include one or more of the following features. Thesequence of frames includes one or more frames detected using a firstgain level and two or more frames detected using a second gain level.The first module is operable to detect motion based on a comparisonbetween frames detected using the second gain level, and the secondmodule is operable to blend a frame detected using the first gain leveland a frame detected using the second gain level. A filter is operableto filter the blended frames to remove noise based, at least in part, onthe gain level of the blended frames. The memory includes a circularbuffer for storing the sequence of frames.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a video acquisition system.

FIG. 2 is a flow diagram of a process for processing video images usinga graphics processor.

FIG. 3 is a schematic diagram of a system for sending data from a camerato a video digitizer.

FIG. 4 is a flow diagram of a process for processing detected images.

FIG. 5 is a block diagram of a temporal filter for reducing noise in asequence of video images.

FIG. 6 is a flow diagram of a process for filtering images in a sequenceof video frames.

FIG. 7 is a schematic diagram of a set of pixels in a detected image.

FIG. 8 is a block diagram of a temporal noise reduction system.

FIG. 9 is a chart showing representative contents of the buffersdescribed in FIG. 8 when the pixel data is in RGB 4:4:4 format.

FIG. 10 is a chart showing representative contents of the buffersdescribed in FIG. 8 when the pixel data is in YUV 4:2:2 format.

FIG. 11 is a block diagram of a dynamic range expansion system for avideo sequence.

FIG. 12 is a graph of a process for identifying appropriate gains foruse by the camera in an expanded dynamic range mode.

FIG. 13 is a flow diagram of a process for performing dynamic rangeexpansion of video images.

FIG. 14 is a graph of one example of a function for determining a valueof an adaptive noise variance parameter (NV).

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a video acquisition system 100 in which adigitizing pipeline 105 is implemented in a host device 110 thatreceives both video data 115 and ancillary data 120 from a camera 125.The camera 125 can include an image sensor, such as a CMOS digital imagesensor or a CCD sensor for detecting images. In addition, the camera 125can perform processing on detected images and/or to adjust settings fordetecting images. For example, the camera 125 can perform functions suchas automatic exposure, white balancing, fixed pattern noise reduction,color processing, sharpness, resizing, and other functions. The camera125 can be a peripheral device that is separate from the host device 110or can be built into the host device 110. In a typical implementation,the host device 110 is a notebook computer or a desktop computer,although the host device 110 can be implemented using other types ofprocessing devices. The camera 125 can have virtually any resolution(e.g., 640×480 or 1200×675).

In the illustrated implementation, the camera 125 sends data to the hostdevice 110 on a USB high-speed connection 130, although otherimplementations can use any type of bus, cable, wireless interface, orother signaling connection. The data sent to the host device 110 fromthe camera 125 includes video data 115 that defines the appearance ofdetected images (e.g., pixel values) and ancillary data 120 that definesother information relating to the detected images (e.g., settings andstates of the image sensor in detecting the images). The ancillary data120 can include, for example, information above and beyond data includedin the Universal Serial Bus Device Class Definition for Video Devices,Revision 1.1, published by the USB Implementers Forum. The video data115 can be in any format, such as YUV format with 4:2:2 subsampling(e.g., using a 2vuy or other compression type) or RGB 4:4:4 format. Forpurposes of this description, the term “YUV” is not limited to true YUVvalues but includes data and values scaled according to Y′UV, Y′CbCr,and Y′PbPr scale factors, where the prime symbol indicates nonlinearity.Similarly, the term “luminance” includes luma, which is a weighted sumof nonlinear (gamma-corrected) R′G′B′ components used to approximatetrue CIE luminance for purposes of video systems.

The ancillary data 120 can also be received from other peripheraldevices and/or components internal or external to the host device 110,including from other modules included in the host device 110. Forexample, the host device 110 can receive temperature information fromtemperature sensors, such as those used to control fans for cooling anotebook, desktop, or other computer system. Ancillary data 120 such astemperature information can be used to determine conditions, states, orsettings that exist at the time or approximately at the time that imagesare detected.

The host device 110 includes a video digitizing pipeline 105 forprocessing data received from the camera 125. The video-digitizingpipeline 105 is typically implemented on a CPU using one or moresoftware modules, such as a QuickTime video digitizer component,available from Apple Computer Inc. of Cupertino, Calif. and otherproviders of QuickTime-compatible video input systems. As illustrated,the video digitizer 105 can communicate with a graphics processing unit(GPU) 140 that is part of the host device 110 or that is communicablycoupled to the host device 110. The GPU 140 can perform some functionsof the video digitizer 105. In particular, video and graphics processingfunctions of the video digitizer 105 can be offloaded to the GPU 140.The GPU 140 is a single-chip microprocessor specifically designed forprocessing three-dimensional graphics data, such as the GeForce productsavailable from NVIDIA Inc. of Santa Clara, Calif., or GPUs availablefrom ATI Technologies Inc. of Markhan, Ontario, or integrated graphicssolutions available from Intel Corporation of Santa Clara, Calif. TheGPU 140 includes a vector-based instruction set and operates toautomatically perform multiple iterations of a single instruction (ormultiple instructions) on multiple data items. For example, the GPU 140operates to automatically perform multiple iterations of a particularinstruction on a set of pixel values and, as such, can performprocessing of graphics data much more efficiently than a CPU.

The CPU communicates with the GPU 140 using a relatively high-speedconnection, such as Peripheral Component Interconnect Express (PCI-E)145. The video digitizer 105 is used to perform auxiliary orsupplemental processing of the data received from the camera 125. Theauxiliary processing can include spatial and/or temporal noisereduction, color processing (e.g., correction or calibration), sharpnessadjustments, resizing, and/or optional effects, such as flipping animage. In some implementations, at least most of the auxiliaryprocessing is performed on the GPU 140, although the CPU can be used toperform some pre- or post-processing, such as formatting, masking,compressing, or encoding. The auxiliary processing that is performed onthe GPU 140 can be performed using pixel shading programs (or pixelshaders), such as fragment programs that comply with Open GLArchitecture Review Board (ARB) standards. ARB fragment programs arewritten in a generic assembly language that can be translated into thenative language of, for example, NVIDIA, ATI, Intel integrated graphicsprocessors, and the like.

The video-digitizing pipeline 105 generates processed video data thatcan be used by a client process 150. The processed video data can betransported to the client process 150 using a memory buffer 155 and/orusing some type of bus, network, wireless interface, and/or otherinterconnection. The client process 150 can reside on the host device110 or on a different device. The processed video data can be usedrelatively immediately from the memory buffer 155 or can be storedrelatively long term on a storage device for later use by the clientprocess 150. The client process 150 can include, for example, any typeof software application designed to receive and/or operate on videodata, such as iChat videoconferencing software, QuickTime multimediasoftware, and iMovie video processing, all available from Apple ComputerInc. of Cupertino, Calif.; EvoCam webcam software available fromEvological Software at www.evological.com; BTV Pro video processingsoftware available at www.bensoftware.com; and other video editingand/or processing software.

FIG. 2 is a flow diagram of a process 200 for processing video imagesusing a graphics processor. Video data and/or ancillary data for asequence of video images is detected by an image sensor and/or by otherperipheral components (205) and received by a computer (210). The videodata generally includes pixel values for the various pixels in eachimage. In some implementations, the video images are generally detectedand received at a rate of about thirty frames per second, although lower(e.g., twelve or twenty five frames per second) or higher (e.g., sixtyframes per second) frame rates can also be used. In some instances,video images can be received at a rate of about thirty frames per secondin normal or bright lighting conditions and at a lower rate (e.g., aboutfifteen frames per second in low lighting conditions. Typically, thedata is received by a CPU.

The received data is forwarded to and processed using a graphicsprocessor to adjust one or more visual characteristics of the videoimages represented by the received data (215). The received data can beprocessed to reduce temporal and/or spatial noise, correct and/orcalibrate color, scale images, and/or adjust sharpness. Temporal noisereduction generally involves using pixel data from one or more precedingor subsequent frames in a sequence of video images to filter apparentlyerroneous deviations in pixel values between temporally separatedimages. Spatial noise reduction generally involves using pixel data fromone or more adjacent pixels in an image to filter apparently erroneouspixel values. Such errors in pixel values are relatively prevalent inlower quality image sensors and/or in low lighting conditions. Thefiltering that is performed does not necessarily eliminate the errors,but it can significantly reduce their prominence.

After processing by the graphics processor, the processed data isforwarded back to the CPU or stored in memory for use by one or moreclient processes running on the CPU and/or on different devices (220).In some implementations, a client process on the CPU can performcompression, encoding, or reformatting of the processed data receivedfrom the graphics processor before forwarding the data along for use byother client processes. In general, by offloading processing of videoframes to the graphics processor, processing resources and/or powerconsumption of the CPU can be saved or used for other purposes.

FIG. 3 is a schematic diagram of a system 300 for sending data from acamera 305 to a video digitizer 310. The data includes video data 315,which defines images detected by the camera, and ancillary data 320,which defines camera settings and/or other information relating to thestate of the camera and/or the environment during or at approximatelythe time at which the images are detected. The ancillary data 320 caninclude, for example, values from a subset of registers included in thecamera that store status and control data, such as RGB (red, blue, andgreen) gain levels (i.e., the amount of gain applied in detecting red,blue, and green components of each pixel), current luminance,time-averaged luminance, and auto-exposure and/or white balance modecontrol. Other information can also be included in the ancillary data320.

In some implementations, data is sent from the camera 305 to the videodigitizer using a USB bus 325, so the data is segmented into packets.When the camera 305 detects a new frame, the camera 305 generates apacket 330 that includes a start of frame (SOF) marker 335, a portion ofthe video data 315, and a portion of the ancillary data 320. Portions ofthe video data 315 and the ancillary data 320 are also inserted intosubsequent packets 330 until the entire frame is transferred. In someimplementations, the end of the frame can be indicated by an end offrame marker. In addition, packets can be generated and sent during theprocess of detecting the image.

Generally, the ancillary data 320 included in a packet 330 correspondsto the video data 315 also included in the packet 330. In some cases, itis not necessary to include ancillary data 320 in every packet 330.Instead, ancillary data 320 can be inserted into packets 330 to theextent there is a change in the ancillary data 320 since the last timeit was included in a packet 330. Accordingly, it may be possible forancillary data 320 associated with one frame to apply for the entireframe or for multiple frames. In other cases, new ancillary data 320 maybe included when there is a change in settings or other information inthe middle of a frame. In another alternative, ancillary data 320 can besent in response to a query from the video digitizer 310, although sucha technique may result in a delay in recognizing updated settings.

On the receiving end, a determination that the complete frame has beenreceived can be determined based on the receipt of a new start of framemarker 335 (or the receipt of an end of frame marker). The video data315 from the packets 330 is extracted into one or more buffers 340associated with the video digitizer 310. Each buffer 340 can store datafor one frame. Thus, the use of multiple buffers 340 allows the videodigitizer 310 to store multiple frames (e.g., to enable processing ofone frame while capturing another). In addition, the video digitizer 310can timestamp each frame that is received by inferring a time from whenthe start of frame marker 335 is received on the USB transport wire 325,which may precede processing and/or further delivery of the processedframe by tens of milliseconds. The ancillary data 320 is provided foruse by processing code 345 of the video digitizer. For example, theancillary data 320 can be used to estimate an amount of noise in thevideo data 315 and/or to control filtering strengths for reducing noise.Dimmer lighting conditions generally need higher gain levels to provideacceptable images. Higher gains also amplify noise, and, as a result,the gain level can be used to estimate the ratio of signal to noise inthe video data 315.

The types of ancillary data 320 that are included in the packets 330and/or that are used by the video digitizer 310 can depend upon the typeof camera 305. In particular, the ancillary data 320 and the processingcode 345 (or parameters associated therewith) can be tuned for the typeof camera 305. The processing code 345 can include instructions forperforming noise reduction, resizing, color correction or calibration,image sharpening, and/or formatting changes. The processing code 345 canbe implemented in a CPU and/or a graphics processor. Once a completeframe is received, the video data 315 for that frame stored in thebuffer 340 is processed using the processing code 345 to generateconverted frames 350, which can be sent to a client device or process.The converted frames 350 can also be stored in one or more buffers, eachof which can store a different converted frame 350. By storing multipleconverted frames 350, the converted frames 350 can be queued fordelivery to the client device or process (e.g., in case the client takestoo much time handling one frame before requesting the next frame). Thetimestamps provided by the video digitizer 310 can be used by the clientto determine the time at which each of the queued frames was detected.

FIG. 4 is a flow diagram of a process 400 for processing detectedimages. Video data for a sequence of video images is detected by animage sensor (405) and received by a computer (410). Settings data forthe image sensor is also received by the computer (415). Typically, thesettings data reflects settings for the image sensor at the time orapproximately the time the video data is detected. The settings dataincludes gain levels, luminance information, and/or other informationrelating to adjustable settings of the image sensor and detectedenvironmental conditions. The video data is processed using the settingsdata to produce converted frames (420). For example, the settings datacan be used as a parameter or variable in one or more processingalgorithms used to convert the video data into converted frames.

FIG. 5 is a block diagram of a motion-adaptive temporal filter 500 forreducing noise in a sequence of video images. Filtering of the videoimages can be performed using known equations that are based on a Kalmanfilter. Frames of video from a camera constitute a current input x(t)and are stored in an input buffer 505. The current input x(t) includesthe pixel values for each pixel in an input image (i.e., the currentframe). A motion history buffer 510 stores weighted motion data for apreceding frame. The weighted motion data serves as a pixel-by-pixelmeasure of motion between successive images in the sequence of videoimages. An output buffer 515 stores a filtered frame (i.e., output pixelvalues) for the preceding frame. A motion detection module 520 serves todetect motion between successive frames based on the data stored in theinput buffer 505, the motion history buffer 510, and the output buffer515. In addition, the motion detection module 520 receives a noisevariance parameter (NV, which can be a function of camera parameters,such as gain, and/or x(t))) and one or more motion threshold parameters(e.g., MTh₀, MTh₁, MTh₂, which may be defined in units of luminance) foruse in determining a filtering strength.

The noise variance parameter is computed as a function of ancillarydata. In some implementations, the ancillary data is received from thecamera or image sensor. In addition, the noise variance parameter foreach pixel can be a function of the brightness level of the pixel. Forexample, the noise variance parameter can be computed as a function ofthe gain level (e.g., RGB gains) used for detecting the input frameand/or the brightness level of each pixel. Alternatively, the noisevariance parameter can be determined using a look-up table based on thegain level and/or the brightness level of each pixel. In general, lowerlighting conditions result in a higher gain level, a higher noise level,and a stronger filter (i.e., more of a contribution from the precedingfiltered output frame). The motion threshold parameter can be apredetermined threshold value that is manually or automatically defined.

The motion detection module 520 generates two outputs. One output is anupdated weighted motion data vector 525 for the current frame. Theupdated weighted motion data vector 525 is based in part on a currentmotion vector CurrentMotion(t). The current motion vector indicates anamount of motion between the current frame and the preceding frame andcan be calculated using any one of a variety of different techniques.For example, motion values for each individual pixel in the currentmotion vector can be calculated based on an absolute difference betweenthe pixel in the preceding filtered frame and the pixel in the currentframe. The difference can be based on a difference in luminance valuesfor corresponding pixels in the two successive frames. In someimplementations, the value for each pixel in the current motion vectorcan be defined as the maximum of the motion values for the pixel andfour of its closest neighbors (i.e., left, right, top, and bottom). Afiltered motion vector M(t) is calculated as:M(t)=CurrentMotion(t)+M(t−1)*(1−K(t−1)),  (1)where the second term of the filtered motion vector is the weightedmotion data for the preceding frame. For an initial frame in a videosequence, the weighted motion data for the preceding frame is equal tozero.

Another output is a filtering parameter K(t), which is calculated as:K(t)=1, for M(t)>MTh ₀ or x(t)>MTh ₂,K(t)=M(t)/(M(t)+NV(CameraParameters, x(t))), for M(t)<MTh.  (2)The CameraParameters value can include one or more parameters orsettings data received from the camera. Alternatively or in addition,the noise variance can be based on ancillary data obtained from othersources, although such a feature is not explicitly included in equation(2). For an initial frame in a video sequence, the filtering parameteris equal to one, and the filter is disabled. The filtering parameterK(t) is used to determine a filtering strength (e.g., 1−K(t)), whichcorresponds to a degree to which a previous filtered output y(t−1) iscombined with the current input x(t) to generate a current filteredoutput y(t). Due to an infinite impulse response (IIR) nature of thefilter, motion blur artifacts may occur and tend to be more visible inbright areas, which is also where noise is less perceptible and asignal-to-noise ratio (SNR) is high. By using of the motion thresholdsMTh₀ and MTh₂, the filter is disabled in the area of relatively highmotion and/or relatively bright luminance. Thus, the estimated amount ofnoise is determined, for example, by the gain level used to detect thecurrent frame and/or by the lighting conditions. The updated weightedmotion data vector 525 is calculated using the filtering parameter K(t)and has a value of:M(t)*(1−K(t)).  (3)The updated weighted motion data vector 525 is stored in the motionhistory buffer 510. In some implementations, instead of calculating andstoring the updated weighted motion data vector 525, the motion historybuffer 510 can store the filtered motion vector M(t) and the filteringparameter K(t) individually; in such a case, the weighted motion datavector 525 can be calculated by the motion detection module 520, asneeded for performing motion detection on the next frame.

FIG. 14 is a graph 1400 of one example of a function for determining avalue of an adaptive noise variance parameter (NV). The vertical axis1405 corresponds to the noise variance value, and the horizontal axis1410 corresponds to the current input x(t). The initial value 1415 ofthe noise variance (e.g., 28, 32, 64, or 92) is determined based on thecamera parameters (or on other ancillary data) and is used for pixelvalues of the current input x(t) that are less than a motion thresholdMTh₁. The noise variance value (as indicated at 1420) decreases linearlyfrom the initial value 1415 to zero for pixel values of the currentinput x(t) between the motion thresholds MTh₁ and MTh₂ (e.g., betweenpixel luminance values of 200 and 230). The noise variance value is zerofor pixel values of the current input x(t) above the motion thresholdMTh₂. Thus, the adaptive noise variance parameter can be defined foreach pixel as:

$\begin{matrix}\begin{matrix}{{{NV} = {NV}_{0}},{{{for}\mspace{14mu}{x(t)}} < {MTh}_{1}},} \\{{= {{NV}_{0}*{( {{MTh}_{2} - {x(t)}} )/( {{MTh}_{2} - {MTh}_{1}} )}}},} \\{{{{for}\mspace{14mu}{MTh}_{1}} < {x(t)} < {MTh}_{2}},} \\{{= 0},{{{for}\mspace{14mu}{x(t)}} > {MTh}_{2}},}\end{matrix} & (4)\end{matrix}$where NV₀ is a function of the camera parameters (e.g., gain for eachpixel) or other ancillary data.

The temporal filter 500 also includes a first order infinite impulseresponse (IIR) filtering module 530. The inputs to the first order IIRfiltering module 530 include the filtering parameter K(t), the currentinput x(t) and the previous filtered output y(t−1). The first order IIRfiltering module 530 calculates a current filtered output y(t) as:y(t)=K(t)*x(t)+(1−K(t))*y(t−1).  (5)The current filtered output y(t) is stored in the output buffer 515 andis provided as a processed or converted frame for use by a clientprocess. The motion history buffer 510 and the output buffer each can beimplemented using a double buffer technique, as described below inconnection with FIG. 8.

In some implementations, other ancillary data can also be used inprocessing frame data. For example, ancillary data relating to operatingtemperature can be used to determine whether the image sensor and/orother components are subject to temperature outside of predeterminedthermal specifications, which can cause some fixed pattern noise in thedetected images. Such fixed pattern noise can show up as red and bluepixels at fixed locations that vary between different sensors and can beexacerbated in low light and high heat. This other ancillary data can beused to modify the noise reduction parameters or algorithms or toimplement additional image processing (e.g., when the sensor is toohot). Such processing can be implemented in the motion detection module520, the first order IIR filtering module 530, or the blending module1145 of FIG. 11, as part of the GPU or another processing component.

FIG. 6 is a flow diagram of a process 600 for filtering images in asequence of video frames. A video frame is detected using an imagesensor (605). Gain levels used by the image sensor are also identified(610). A noise variance is determined based on the gain levels (615).The noise variance defines an estimated noise level for the video frame.A degree of motion is detected between the video frame and a precedingframe in the sequence of video frames (620). In some implementations,the degree of motion can be based on a comparison of the video framewith an unprocessed version of the preceding frame. In otherimplementations, the degree of motion can be based on a comparison ofthe video frame with a processed or filtered version of the precedingframe. In some implementations, the degree of motion can further includea contribution of detected motion from one or more preceding frames(i.e., the degree of motion can be filtered based on motion betweenpreceding frames). A filtering strength is calculated based on the noisevariance and the detected degree of motion (625). The video frame isthen filtered using the preceding frame and based on the calculatedfiltering strength (630). In some implementations, the preceding frameused in filtering the video frame can be an unprocessed or unfilteredversion, although the preceding frame used in such filtering istypically a processed or filtered version of the preceding frame.

FIG. 7 is a schematic diagram of a set 700 of pixels in a detectedimage. A degree of motion between the detected image and a precedingimage is determined on a pixel-by-pixel basis. Accordingly, for aparticular pixel 705, motion can be estimated, for example, by comparingthe luminance of the pixel 705 with the luminance of a correspondingpixel in the preceding image. If the luminance remains unchanged, it canbe determined that no motion is present. If there is a change inluminance above a predetermined motion threshold, it can be determinedthat motion is present for the pixel 705. In some implementations, thedegree of motion for the particular pixel 705 can be estimated by theabsolute difference in the luminance values between the consecutiveframes. In addition, in some implementations, the degree of motion forthe particular pixel 705 can be estimated as a maximum absolutedifference in luminance for the particular pixel 705 and its fourneighboring pixels 710, 715, 720, and 725 to the left, right, top, andbottom. Accordingly, if the pixel 715 to the right of the particularpixel 705 has a larger absolute difference in luminance betweenconsecutive frames than does the particular pixel 705 or its otherneighboring pixels 710, 720, and 725, the motion value assigned to theparticular pixel 705 is the calculated motion of the pixel 715 to theright. In some cases, the degree of motion may be based on the maximumamong a lesser or greater number of neighboring pixels.

FIG. 8 is a block diagram of a temporal noise reduction system 800. Thesystem 800 can be implemented using texture buffers of a GPU and usingpixel-shading programs (e.g., ARB fragment programs) running on the GPU.In some implementations, the system 800 can use other types ofprocessors, hardware, or environments. The temporal noise reductionsystem 800 includes multiple buffers, including an unprocessed framebuffer 805, first and second processed frame buffers 810 and 815, andfirst and second motion history buffers 820 and 825. The processed framebuffers 810 and 815 and the motion history buffers 820 and 825 aredepicted twice in FIG. 8 for purposes of convenience, but it will beunderstood that the duplicate buffers reflect the same logical orphysical buffer.

The temporal noise reduction system 800 also includes first, second, andthird binary multiplexers 830, 835, and 840 and first and seconddemultiplexers 845 and 850. Frames in a video sequence are successivelyprocessed by the temporal noise reduction system 800. Each successiveframe is either an even frame or an odd frame. The state of themultiplexers 830, 835, and 840 and demultiplexers 845 and 850 iscontrolled by a binary variable Even 855 or its complement 860, whichcan be provided by an inverter 865. For even frames, the value of thebinary variable Even 855 is 1, and, for odd frames, the value of theEven variable 855 is 0. Each of the multiplexers 830, 835, and 840 anddemultiplexers 845 and 850 uses one of its two inputs or outputs whenthe Even variable 855 has a first value (e.g., 1) and the other of itstwo inputs or outputs when the Even variable 855 has a second value(e.g., 0).

The temporal noise reduction system 800 includes two pixel-shadingprograms—a motion detection fragment program 870 (see motion detectionmodule 520 of FIG. 5) and a first order IIR filter fragment program 875(see first order IIR filtering module 530 of FIG. 5). The multiplexers830, 835, and 840 and demultiplexers 845 and 850, as well as the pixelshading programs 870 and 875 can be implemented in the logic of a GPU(e.g., as software instructions).

Frames are received by a host device from a camera and are forwarded oneat a time to the unprocessed frame buffer 805, where each frame istemporarily stored for processing by the temporal noise reduction system800. The frame data stored in the unprocessed frame buffer 805 isprovided as one input to the motion detection fragment program 870.Assuming, for purposes of illustration, the current frame in theunprocessed frame buffer 805 is an even frame, the first multiplexer 830provides the frame data stored in the second processed frame buffer 815as another input to the motion detection fragment program 870. Thesecond multiplexer 835 provides the motion data stored in the secondmotion history buffer 825 as an input to the motion detection fragmentprogram 870. The motion detection fragment program 870 uses the inputdata to calculate motion data on a pixel-by-pixel basis and to calculatea parameter relating to temporal filtering weight, as discussed above inconnection with the motion detection module 520 of FIG. 5. Thecalculated motion data and filter weighting parameter are sent to thefirst demultiplexer 845. Based on the Even complement variable 860, thefirst demultiplexer 845 causes the received data to be stored in thefirst motion history buffer 820.

The frame data stored in the unprocessed frame buffer 805 is alsoprovided as one input to the first order IIR filter fragment program875, along with the frame data stored in the second processed framebuffer 815. Based on the Even variable 855 and the operation of theinverter 865, the third multiplexer 840 provides the motion data storedin the first motion history buffer 820 (i.e., the output of the motiondetection fragment program 870 discussed above) as another input to thefirst order IIR filter fragment program 875. The first order IIR filterfragment program 875 uses the input data to calculate processed framedata for the current frame, which the second demultiplexer 850 stores inthe first processed frame buffer 810 under the control of the Evencomplement variable 860. In addition, the processed frame data is sentto the host device to provide a processed frame for use in one or moreclient processes.

The next frame, which is an odd frame, that is received by the temporalnoise reduction system 800 is stored in the unprocessed frame buffer805. The motion detection fragment program 870 receives as inputs theframe data from the unprocessed frame buffer 805, the processed framedata from the first processed frame buffer 810, and the motion data fromthe first motion history buffer 820, the latter two of which werepopulated by the previous iteration of the fragment programs 870 and875. The motion detection fragment program 870 generates motion datathat is stored in the second motion history buffer 825. The first orderIIR filter fragment program 875 receives as inputs the frame data fromthe unprocessed frame buffer 805, the processed frame data from thefirst processed frame buffer 810, and the just-stored motion data fromthe second motion history buffer 825. The first order IIR filterfragment program 875 generates processed frame data that is stored inthe second processed frame buffer 815 and delivered to the host devicefor use in one or more client processes. By implementing the temporalnoise reduction system 800 in a GPU using fragment programs and texturebuffers, processing of video sequences can be efficiently performed.

GPU processing techniques can also be used for other types of videoprocessing. For example, GPU processing can be used to address potentialproblems with fixed pattern noise in the images. Because fixed patternnoise is particularly prevalent in areas with low lighting conditions,as discussed above, less gain can be applied to red and blue pixelcomponents in relatively dark image areas (i.e., areas with lowluminance values). This effect can be applied during gamma processing,which involves modifying the luminance characteristics to correct thebrightness profile of an image. In one implementation, single-channelgamma correction is applied to uniformly modify the luminance of darkareas of an image (e.g., using a look-up table to apply a gamma curve tothe input luminance value), and three-channel gamma correction on R, G,and B channels is applied to other areas (e.g., by applying gamma to thethree components in the RGB color space). The three-channel gammacorrection provides more accurate colors, while the single-channel gammacorrection can be performed faster and/or more efficiently.

In another implementation, to achieve improved color accuracy, bothsingle-channel and three-channel gamma correction are determined, and anon-linear interpolation between the two is performed. In addition, theinterpolation can be performed using, for example, a fourth orderpolynomial (e.g., 1−(1−x)⁴, for 0x<1, where x represents the inputluminance value) to rapidly favor the three-channel gamma correction forhigher luminance values to provide more accurate colors and less fixedpattern noise. In some implementations, the non-linear interpolationfunction can also include an offset to avoid restricting theinterpolation range and to ensure that the interpolation includes somecontribution from the three-channel gamma correction calculation. Forexample, twenty five percent (25%) or fifty percent (50%) of thethree-channel gamma value can be blended with the single-channel gammavalue provided by the fourth order polynomial. This blending can beperformed using an interpolation function:g(x)=b+(1−b)*f(x),  (6)where f(x) is a polynomial function, such as f(x)=1−(1−x)⁴, for 0<x<1,and g(x)=0.25+0.75*f(x), for example. The function g(x) is thenmultiplied by the single-channel-gamma value and added to (1−g(x))multiplied by the three-channel-gamma value to produce an interpolatedgamma correction. The use of a non-linear interpolation function canprovide a rapid convergence toward three-channel-gamma correction asluminance increases from zero (or some other relatively low luminancevalue). This technique avoids an excessive loss of color when thesingle-channel gamma correction calculation is favored for low luminancevalues, but can slightly increase the visibility of the fixed patternnoise depending on the amount of offset and how rapid the convergenceis.

FIG. 9 is a chart 900 showing representative contents of the buffersdescribed in FIG. 8 when the pixel data is in RGB 4:4:4 format.Typically, the texture buffers of a GPU include space for four datavalues for each pixel entry. In some implementations of the temporalnoise reduction system 800 of FIG. 8, the buffers that store unprocessedand processed frame data 905 (i.e., the unprocessed frame buffer 805 andthe first and second processed frame buffers 810 and 815) include a redvalue R_(n), a green value G_(n), a blue value B_(n), and an extraparameter value A_(n) for each pixel n 915. The extra parameter valuesA_(n) are ignored for purposes of temporal filtering. Each combined RGBvalue 920 constitutes a vector-based pixel value.

The buffers that store motion data 910 (i.e., the first and secondmotion history buffers 820 and 825) include filtering weights K_(rn),K_(gn), and K_(bn) corresponding to each of the red, green, and bluecomponents or values and a motion value M_(n) for each pixel n 915. As aresult, filtering weights K_(rn), K_(gn), and K_(bn) and a motion valueM_(n) for each pixel n 915 are stored in a single entry 925 of themotion history buffers 820 and 825. The motion value M_(n) for eachpixel is determined using luminance information (e.g., Y in the YUVspace) for the pixel based on a comparison with luminance informationfor a preceding frame. A luminance value can be computed in a singleinstruction based on the red, green and blue values from the RGB 4:4:4format. By calculating filtering weights per component (i.e.,calculating K_(rn), K_(gn), and K_(bn)), an independent noise variancecan be determined for each component, and noise filtering can beindependently tuned for each component. Thus, the motion detectionmodule 520 of FIG. 5 and/or the motion detection fragment program 870 ofFIG. 8 can be used to calculate per-component filter weights K_(r),K_(g), and K_(b). Likewise, the first order IIR filtering module 530 ofFIG. 5 and the first order IIR filter fragment program 875 of FIG. 8 canbe used to perform filtering on a per-component basis.

FIG. 10 is a chart 1000 showing representative contents of the buffersdescribed in FIG. 8 when the pixel data is in YUV 4:2:2 format. Usingthe YUV 4:2:2 format allows the buffers to be two times smaller than inthe RGB 4:4:4 case. This is because every two pixels share chrominancevalues U and V. In other words, while each pixel has a unique luminancevalue Y, each pair of pixels use the same chrominance values U and V.Thus, in some implementations of the temporal noise reduction system 800of FIG. 8, the buffers that store unprocessed and processed frame data1005 (i.e., the unprocessed frame buffer 805 and the first and secondprocessed frame buffers 810 and 815) include chrominance values U_(n)and V_(n) and two luminance values Y_(2n) and Y_(2n+1) for each pixeldata entry 1015. The frame data 1005 for each pixel entry 1015, however,defines pixel values for two pixels (at 1020). For example, a firstpixel includes chrominance values U₁ and V₁ and a luminance value Y₁,and a second pixel includes the same chrominance values U₁ and V₁ but adifferent luminance value Y₂. Similarly, a third pixel includeschrominance values U₂ and V₂ and a luminance value Y₃, and a fourthpixel includes the same chrominance values U₂ and V₂ but a differentluminance value Y₄. The combination of a luminance value Y_(2n) orY_(2n+1) and the corresponding chrominance values U_(n) and V_(n)constitutes a vector-based pixel value.

The buffers that store motion data 1010 (i.e., the first and secondmotion history buffers 820 and 825) include two filtering parametersK_(2n) and K_(2n+1), one for each of the pixels that share chrominancevalues, and two motion values M_(2n) and M_(2n+1), one for each of thepixels that share chrominance values. Thus, the motion data 1010 foreach pixel entry 1015 stores filtering parameters and motion values fortwo pixels (at 1025). The motion value M for each pixel is determinedusing the available luminance information (i.e., the luminance value Y)for the pixel. In the YUV 4:2:2 case, filtering weights are calculatedper pixel based, for example, on the maximum of the red, blue, and greengain values used in detecting the image, as determined from ancillarydata received from the camera by the host device. Noise filtering canthen be performed for the luminance component (e.g., Y₁) of each pixelusing the corresponding filtering weight (e.g., K₁) and for thechrominance components (e.g., U₁ and V₁) using one of the twocorresponding filtering weights (e.g., Minimum(K₁, K₂)). Pixel formatsother than RGB 4:4:4 and YUV 4:2:2, such as YUV 4:4:4 or YUV 4:1:1, canalso be used in connection with one or more of the described techniques.

FIG. 11 is a block diagram of a dynamic range expansion system 1100 fora video sequence. The dynamic range expansion system 1100 can beimplemented, for example, at least in part using software running on acomputer or a GPU. In general, the system 1100 performs dynamic rangeexpansion by combining frames detected using two or more different gainlevels. The illustrated example and the following description depict adynamic range expansion technique using two gain levels, althoughimplementations can use more than two gain levels. Each gain level isanalogous to an exposure level on a conventional camera—a higher gaincorresponds to a longer exposure, while a lower gain corresponds to ashorter exposure. With a high gain, additional details of areas with lowlight can be obtained but areas with greater light can become washedout. A low gain may enable greater detail in the higher lighting areasbut low light areas lack detail. Dynamic range expansion allows detailsfrom both a high gain image and a low gain image to be combined in asingle image.

The dynamic range expansion system 1100 includes an interface 1105 forcommunicating with a camera or image sensor and a circular buffer 1110that, in the illustrated implementation, includes three frame buffers1115, 1120, and 1125. The dynamic range expansion system 1100 alsoincludes a motion detection module 1140 for detecting motion betweensuccessive received frames and a blending module 1145 for performingblending or gain compensation on received frames. A first order temporalfilter 1150 is used to perform temporal noise reduction on blended orgain compensated pixels.

The interface 1105 can be used to request and/or receive settings data.For example, the interface 1105 receives video parameters (as indicatedat 1130), such as an auto-exposure gain level, a gain level used todetect low gain frames, a gain level used to detect high gain frames,and an average luminance. The interface 1105 can also instruct thecamera to switch modes by sending requested settings to the camera (asindicated at 1135). In general, two camera gain modes are used forperforming dynamic range expansion—an auto-exposure mode and an expandeddynamic range mode. In the auto-exposure mode, the camera determines anappropriate gain level based on environmental lighting conditions (e.g.,to achieve a predetermined image quality or an average luminance for theimage or a portion of the image). In the expanded dynamic range mode,the camera switches between a high gain (e.g., higher than the gainlevel identified in the auto-exposure mode) and a low gain (e.g., lowerthan the auto-exposure mode gain level) to detect each successive,temporally spaced frame. Thus, even frames are detected using a highgain and odd frames are detected using a low gain, or vice versa. Insome implementations, the camera determines the gain values for use inthe expanded dynamic range mode, while in other implementations the gainvalues can be determined by the dynamic range expansion system 1100 andsent to the camera (as indicated at 1135).

The circular buffer 1110 stores frame data received from the camera orimage sensor. Because each successive frame is detected usingalternating gain values, each sequential frame buffer 1115, 1120, or1125 includes video data for a frame detected with a different gain thanthe adjacent frame buffer or buffers in the sequence. As each new frameis received, the frames in the circular buffer 1110 are shifted to thenext frame buffer 1120 or 1125 or discarded (in the case of the framestored in the third frame buffer 1125), and the new frame is inserted inthe first frame buffer 1115. In the illustrated example, the currentframe X(t) stored in the first frame buffer 1115 includes frame datadetected using a low gain level, the preceding frame X(t−1) stored inthe second frame buffer 1120 includes frame data detected using a highgain level, and the next preceding frame X(t−2) stored in the thirdframe buffer 1125 includes frame data detected using the low gain level.When the next frame X(t+1), which will be detected using the high gainlevel, arrives, it will be stored in the first frame buffer 1115, andthe original contents of the first frame buffer 1115 and the secondframe buffer 1120 will be shifted, respectively, to the second framebuffer 1120 and the third frame buffer 1125. Alternatively, the contentof the buffers is not moved between the buffers; instead, shiftingbetween the buffers is performed by manipulating buffer pointers.

The motion detection module 1140 detects motion between successiveframes having the same gain level to generate a motion indicator M(t).Motion is determined on a per-pixel basis based, for example, on anabsolute difference in luminance values for corresponding pixels in thesuccessive frames. The motion indicator M(t) can be determined in thesame manner as or in a different way than the filtered motion vectorM(t) or the current motion vector CurrentMotion(t) discussed inconnection with FIG. 5. In the illustrated example, motion is identifiedbetween the frame X(t) stored in the first frame buffer 1115 and theframe X(t−2) stored in the third frame buffer 1125. In an alternativeimplementation, motion can be identified between frames detected usingdifferent gain levels (i.e., between the frame X(t) stored in the firstframe buffer 1115 and the frame X(t−1) stored in the second frame buffer1120). For example, each frame can be normalized by performing gaincompensation (e.g., to remove the effects of the gain applied indetecting the image) before identifying per-pixel motion. Othertechniques for detecting motion can also be used.

The blending module 1145 performs blending of pixels from framesdetected using different gain levels. Typically, blending is performedonly on pixels for which motion is not identified (e.g., to avoidblending pixels that do not correspond to the same object). The blendingmodule 1145 generates a current frame with expanded dynamic range Y(t)by combining the frame data from the first frame buffer 1115 and thesecond frame buffer 1120 according to:Y(t)=[(K1*X(t)/gain_low)+(K2*X(t−1)/gain_high)]/(K1+K2),for M(t)<MTh,X(t)/gain_low, for M(t)>MTh,  (7)for the case where the current frame X(t) is detected using the low gainlevel, where gain_low and gain_high define the gain factors used todetect the images (e.g., gain_low =0.5 and gain_high=1.5); K1 and K2 areblending factors for low gain images and high gain images, respectively;and MTh is a motion threshold. For pixels that are identified as havingmotion above a threshold level, gain compensation is applied tonormalize the pixel value without performing blending (and thus withoutexpanding the dynamic range of such pixels). In some implementations,such as when more than two gain levels are used to detect images,blending may involve additional frames.

In the case where the current frame X(t) is detected using the high gainlevel, the expanded dynamic range frame Y(t) can be calculated accordingto:Y(t)=[(K1*X(t−1)/gain_low)+(K2*X(t)/gain_high)]/(K+1+K2),for M(t)<MTh,X(t)/gain_high, for M(t)>MTh.  (8)The value of the blending factors or parameters K1 and K2 define arelative contribution of each pixel from each image and can becalculated according to:Kn=128−abs(128−LumaValue),  (9)where n=1 for low gain frames, n=2 for high gain frames, and LumaValueis the luminance value for the pixel in the high gain or low gain imageand where luminance values fall within a range of 0 to 255. Otheralgorithms or techniques for determining blending factors can also beused. If the camera response to changes in gain is linear, gaincompensation can be achieved by applying linear scaling, as in the gaincompensation portions of the equations above. In some cases, the cameraresponse may be non-linear, in which case a look-up table (LUT) can beused to identify appropriate scaling (e.g., instead of using thegain_low and gain_high values). The values of an inverse gain look-uptable can be provided, for example, by the interface 1105 or stored in amemory associated with the blending module 1145.

The expanded dynamic range frame Y(t) is then processed by the firstorder temporal filter 1150 to perform temporal noise reduction onblended or gain compensated pixels. The first order temporal filter 1150can be implemented using the temporal filter 500 of FIG. 5 and/or thetemporal noise reduction system 800 of FIG. 8. Thus, the expandeddynamic range frame Y(t) can correspond to the input frame x(t)illustrated in and described with reference to FIG. 5. The first ordertemporal filter 1150 generates a filtered video output.

FIG. 12 is a graph 1200 of a process for identifying appropriate gainsfor use by the camera in an expanded dynamic range mode. The graph 1200represents gain level on the vertical axis 1205 and time on thehorizontal axis 1210. At an initial time t₀, the camera is placed in anauto-exposure mode (AE) in which the camera detects lighting conditionsand determines an appropriate gain level for use in detecting images.Generally, the gain level identified in the auto-exposure mode assumesthat all frames are detected using the same gain level, in contrast toan expanded dynamic range mode. In the example illustrated in the graph1200, time t₀ falls within a period 1230 of low lighting conditions, andthe auto-exposure gain level is set at a first gain level 1215. Afterdetermining the appropriate gain level, the camera is switched into theexpanded dynamic range mode (EDR) at time t₁. Control over the mode canbe handled, for example, by the interface 1105 of the dynamic rangeexpansion system 1100 of FIG. 11.

In the expanded dynamic range mode, the gain level is alternated betweena high gain and a low gain for detecting each successive frame. Forexample, odd frames are detected (as indicated at 1220) using a highgain of two times the first gain level 1215, and even frames aredetected (as indicated at 1225) using a low gain of one-half the firstgain level 1215. The high gain and low gain levels can be determined bythe camera or can be assigned by the dynamic range expansion system 1100through the interface 1105.

At time t₂, the lighting conditions change (e.g., someone flips on alight switch), and a period 1235 of bright lighting conditions begins.This change is detected, for example, by the dynamic range expansionsystem 1100 through the interface 1105 based on a time-averagedluminance parameter received from the camera or calculated by thedynamic range expansion system 1100. Generally, minor fluctuations inlighting conditions do not have a noticeable impact on the time-averagedluminance. If the time-averaged luminance changes by more than somerelatively small percentage or absolute amount from the luminance leveldetected during the last auto-exposure mode, it can be determined thatthe lighting conditions have changed. Accordingly, at time t₃, thecamera is placed back in the auto-exposure mode (e.g., under the controlof the dynamic range expansion system 1100) to identify a second gainlevel 1240 for use during the period 1235 of bright lighting conditions.

At time t₄, after the second gain level 1240 is determined, the camerais again switched into the expanded dynamic range mode, and new highgain and low gain levels are calculated to allow the gain levels toadapt to the new lighting conditions. For example, odd frames aredetected (as indicated at 1245) using a high gain of two times thesecond gain level 1240, and even frames are detected (as indicated at1250) using a low gain of one-half the second gain level 1240.Additional mode changes and determinations of appropriate gain levelscan be performed if and when lighting conditions change.

FIG. 13 is a flow diagram of a process 1300 for performing dynamic rangeexpansion of video images. An image sensor detects one or more videostill images (1305), and lighting conditions for the detected images aredetermined (1310). The images can be detected in an auto-exposure mode,for example, and lighting conditions can be determined explicitly bymeasuring light levels or implicitly by identifying a gain level neededto achieve an acceptable image quality. Based on the lightingconditions, two or more gain levels for alternately using in detectingimages are calculated or identified (1315). The image sensor detectsadditional video still images using the two or more gain levels (1320).For example, odd frames can be detected using a low gain level and evenframes can be detected using a high gain level. Alternatively, fourdifferent gain levels (e.g., two different high gain levels and twodifferent low gain levels) can be used in a repeating sequence to detectframes to further increase dynamic range.

A determination is made as to whether motion between different ones ofthe additional video still images is present (1325). The motiondetermination is typically made for purposes of determining whetherinformation associated with a pixel in one image can be used to providereliable information about a corresponding pixel in another image (e.g.,for purposes of dynamic range expansion, filtering, or colorcorrection). The motion determination is generally made on apixel-by-pixel basis, although in some cases motion can be detected forblocks of pixels. A motion value for each pixel can also be assignedbased on the motion values of one or more adjacent or nearby pixels(e.g., using a maximum or average value function). In someimplementations, motion is determined to be present for a pixel at aparticular pixel address based on whether there is a change in theluminance (and/or other characteristics, such as color, even if theluminance remains unchanged) of the pixel at the particular pixeladdress between the images.

In other implementations, additional processing may be performed toidentify relative motion of objects in the images (e.g., an object inthe foreground, such as a vehicle, moving relative to the background)and/or consolidated motion of the entire image (e.g., through panning ofthe camera). In such cases, it is possible to identify correspondingpixels even though they are located in a different location within theimage. Blending, temporal filtering, and/or other processing may beselectively performed on the different images in which at least some ofthe pixels are offset between the different images. For example,blending can be performed between a block of pixels in one image and acorresponding but offset block of pixels in another image. In someimplementations, such as when the frame rate is sufficiently high toeffectively avoid noticeable blurring, it is possible to omit motiondetection.

If it is determined that motion is not present between the images (orportions thereof), the images (or portions thereof) are blended (1330).Typically, blending is performed on a per-pixel basis for pixels inwhich motion is not present. As discussed above, however, the motiondetermination can be relative to other surrounding pixels (e.g., motionwith respect to corresponding pixels in different images is determinednot to be present if a block of pixels containing the correspondingpixels moves as a block). As such, a pixel from one image can be blendedwith a corresponding pixel from a different image, even though thecorresponding pixel may have a different pixel address (e.g., because ofmovement of the block of pixels). Blending can be used to provide anexpanded dynamic range for a resulting blended image.

If it is determined that motion is present between images (or portionsthereof), the current image (or portions thereof) is gain compensated toproduce a normalized image (1335). As with blending, gain compensationis typically performed on a per-pixel basis for pixels in which motionis present. For example, if motion is determined to be present betweenimages for a particular pixel, the pixel is multiplied by an inverse ofthe gain level used to detect the image to produce a gain-compensatedimage that has approximately the same appearance as if the frame hadbeen detected in an auto-exposure mode.

Each blended and/or gain-compensated image is filtered to remove noise(1340). For example, frames can be processed using a temporal filter. Insome implementations, other types of filtering can be used, such asspatial filtering. The filtered image is a processed frame that has anexpanded dynamic range and reduced noise. The processed frame can beused to replace one of the images from a sequence of the additionalvideo still images for use in subsequent video display or processing.

After each frame or group of frames, a determination is made as towhether lighting conditions have changed (1345). This determination canbe based, for example, on whether an average luminance of a sequence ofimages has changed above a certain amount or is changing at a sufficientrate. If so, the process 1300 returns to detecting one or more videostill images (1305) and determining lighting conditions for the detectedimages (1310). This determination can then be used to identify new gainlevels for use in detecting additional images. If the lightingconditions have not changed, the process 1300 continues detectingadditional video still images using the two or more gain levels (1320).

The invention and one or more of the functional operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuralmeans disclosed in this specification and structural equivalentsthereof, or in combinations of them. The invention can be implemented asone or more computer program products, i.e., one or more computerprograms tangibly embodied in an information carrier, e.g., in a machinereadable storage device or in a propagated signal, for execution by, orto control the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. A computerprogram (also known as a program, software, software application, orcode) can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program does not necessarily correspond to a file. A programcan be stored in a portion of a file that holds other programs or data,in a single file dedicated to the program in question, or in multiplecoordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to beexecuted on one computer or on multiple computers at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

Some processes and logic flows described in this specification,including the method steps of the invention, can be performed by one ormore programmable processors executing one or more computer programs toperform functions of the invention by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus of the invention can be implemented as, specialpurpose logic circuitry, e.g., an FPGA (field programmable gate array)or an ASIC (application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer, includinggraphics processors, such as a GPU. Generally, the processor willreceive instructions and data from a read only memory or a random accessmemory or both. The essential elements of a computer are a processor forexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. Information carriers suitablefor embodying computer program instructions and data include all formsof non volatile memory, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, the invention can be implementedon a computer having a display device, e.g., a CRT (cathode ray tube) orLCD (liquid crystal display) monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user can provide input to the computer. Other kinds ofdevices can be used to provide for interaction with a user as well; forexample, feedback provided to the user can be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including acoustic,speech, or tactile input.

The invention can be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation of the invention, or any combination of such back-end,middleware, or front-end components. The components of the system can beinterconnected by any form or medium of digital data communication,e.g., a communication network. Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”),e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,certain operations of the techniques can be performed in parallel or ina different order than is depicted and described. In addition, thedescribed techniques can be used in connection with performing othertypes of processing on video sequences. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method comprising: receiving captured videodata from an image sensor, the video data defining a sequence of framesincluding a first frame and a second frame detected after the firstframe, with each frame including pixel data for a plurality of pixels;estimating a visual noise level in at least the second frame; detectinga degree of motion between the first frame and the second frame; andfiltering the second frame by modifying pixel data for the second framebased on pixel data from the first frame using a filtering parameterthat defines a degree of the modification, wherein the filteringparameter is determined based on the estimated visual noise level forthe second frame and the degree of motion between the first frame andthe second frame.
 2. The method of claim 1 wherein the visual noiselevel is estimated based on ancillary data identified approximatelyconcurrently with capture of video data for the second frame, theancillary data including at least one parameter for the image sensor. 3.The method of claim 2 wherein the ancillary data comprises a temperaturein a vicinity of the image sensor.
 4. The method of claim 2 wherein theancillary data comprises settings data for the image sensor, thesettings data indicating at least one adjustable setting of the imagesensor used for sensing at least a portion of the video data.
 5. Themethod of claim 1 wherein the first frame comprises a frame having pixeldata that is filtered using pixel data from at least one frame detectedbefore the pixel data for the first frame.
 6. The method of claim 1further comprising modifying luminance characteristics differentlybetween different color channels.
 7. The method of claim 1 whereinfiltering the second frame by modifying pixel data for the second framebased on pixel data from the first frame is performed using differentfiltering parameters for each of a plurality of different colorchannels.
 8. The method of claim 1 further comprising: detecting a blockof pixels in the second frame corresponding to a block of pixels in adifferent location of the first frame; and filtering the block of pixelsin the second frame based on the corresponding block of pixels in thefirst frame.
 9. The method of claim 1 wherein filtering the second frameis perfor med using a graphics processing unit.
 10. An article ofmanufacture comprising a computer-readable non-transitory storage mediumstoring instructions for causing data processing apparatus to: receiveraw video data captured by a camera system, the raw video data defininga sequence of consecutive frames, with each frame including pixel datafor a plurality of pixels; and filter at least some of the frames in thesequence of consecutive frames by modifying pixel data for a selectedframe based on pixel data from at least one preceding frame, wherein adegree of modification is determined based on an estimated visual noiselevel for the selected frame and a degree of motion between the selectedframe and at least one preceding frame.
 11. The article of claim 10wherein the computer-readable non-transitory storage medium storesinstructions for causing data processing apparatus to estimate a visualnoise level in each frame of the sequence of consecutive frames.
 12. Thearticle of claim 11 wherein the computer-readable non-transitory storagemedium stores instructions for causing data processing apparatus toreceive ancillary data identified approximately concurrently with theraw video data by the image sensor, the ancillary data indicating atleast one characteristic ancillary to the video data, wherein the visualnoise level in each frame of the sequence of consecutive frames isestimated based on the ancillary data.
 13. The article of claim 12 theancillary data comprises settings data for the camera system, thesettings data indicating at least one adjustable setting of the camerasystem used for sensing at least a portion of the video data.
 14. Thearticle of claim 10 wherein the computer-readable non-transitory storagemedium stores instructions for causing data processing apparatus todetect a degree of motion between consecutive frames in the sequence ofconsecutive frames.
 15. The article of claim 10 wherein thecomputer-readable non-transitory storage medium stores instructions forcausing data processing apparatus to modify luminance characteristicsdifferently between different color channels.
 16. The article of claim10 wherein filtering the second frame by modifying pixel data for thesecond frame based on pixel data from the first frame is performed usingdifferent filtering parameters for each of a plurality of differentcolor channels.
 17. The article of claim 10 wherein thecomputer-readable non-transitory storage medium stores instructions forcausing data processing apparatus to: detect a block of pixels in thesecond frame corresponding to a block of pixels in a different locationof the first frame; and filter the block of pixels in the second framebased on the corresponding block of pixels in the first frame.
 18. Asystem for producing a sequence of converted video frames comprising: amemory adapted to store raw video data defining a sequence of framesfrom a camera system, with each frame including pixel data for aplurality of pixels; and a module operable to produce a modifiedsequence of video frames by modifying pixel data for a selected framefrom the sequence of frames based on pixel data from at least onepreceding frame, wherein a degree of modification is determined based onan estimated visual noise level for the selected frame and a degree ofmotion between the selected frame and at least one preceding frame. 19.The system of claim 18 wherein the memory is further adapted to storeancillary data defining at least one characteristic ancillary to thevideo data, wherein the visual noise level for the selected frame isestimated based on the ancillary data.
 20. The system of claim 19wherein the ancillary data includes at least a gain level used by thecamera system for detecting the selected frame.
 21. The system of claim18 wherein the module is implemented on a graphics processing unit. 22.The system of claim 18 further comprising at least one module adaptedto: detect a block of pixels in the selected frame corresponding to ablock of pixels in a different location of the at least one precedingframe; and filtering the block of pixels in the selected frame based onthe corresponding block of pixels in the at least one preceding frame.23. The method of claim 1, further comprising encoding the video datasubsequent to the filtering of the video data.